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Executive  Summary 


Many  Department  of  Defense  (DoD)  support  organizations  are  financed  with  re¬ 
volving  funds  to  take  advantage  of  their  flexibility.  However,  in  the  appropria- 
tions-oriented  Defense  Programming  Review,  there  has  been  little  visibility  of  the 
internal  financial  operating  plans  of  those  support  organizations  and  of  the  total 
funding  that  their  customers  plan  to  spend  on  them. 

DoD  is  interested  in  developing  performance  metrics  applicable  to  program 
analysis  of  defense  support  business  areas.  We  identify  key  indicators  geared  to¬ 
ward  this  purpose. 

Our  research  and  findings  focus  on  three  specific  performance  metric  areas:  out¬ 
put,  cost,  and  quality.  In  performing  this  research,  we  concentrated  on  identifying 
potential  metrics  in  these  three  specific  areas  and  evaluating  the  existing  support¬ 
ing  data  and  its  validity.  We  also  observed  the  collected  data,  the  metrics,  and  the 
potential  indications  of  those  data. 

To  minimize  the  disruption  to  Office  of  the  Secretary  of  Defense  activities,  agen¬ 
cies,  and  the  Services  during  the  course  of  this  task,  it  was  decided  that  new  data 
reporting  requirements  would  be  minimized.  Therefore,  we  used  existing  data 
elements  to  the  maximum  extent  possible.  This  constraint  affects  both  historical 
and  projected  data. 

Our  findings  and  conclusions  pertain  to  the  two  broad  purposes  of  this  task:  (1) 
identifying  potential  metrics  and  evaluating  the  supporting  data;  and  (2)  observing 
the  collected  data,  the  associated  metrics,  and  the  potential  indications  of  the  met¬ 
rics.  The  conclusions  also  include  important  cautions  on  the  use  of  the  resulting 
metrics. 

We  believe  that  the  set  of  metrics  areas  (i.e.,  output,  cost,  and  quality)  we  devel¬ 
oped  can  be  useful  in  conducting  program  analysis  of  the  Depot  Maintenance 
business  area.  The  metrics  can  be  linked  with  major  success  factors  associated 
with  the  DoD’s  strategic  goals  and  missions.  These  metrics,  taken  together,  form 


a  reasonably  good  picture  of  resource-related  trends  in  output  and  cost  perform¬ 
ance. 

Sufficient  data  to  construct  the  output  and  cost  metrics  are  available  in  the  Budget 
Estimate  Submissions  (BESs).  Because  of  changes,  to  include  transfers  of  mis¬ 
sions  between  organizations,  care  must  be  exercised  when  extracting  data  to  en¬ 
sure  a  consistent  baseline  is  being  used. 

Quality  performance  data  are  in  a  substantial  shortfall.  Not  all  Services  adequately 
reported  quality  data  in  the  Depot  Maintenance  Operations  Indicators,  and  the 
BES  does  not  include  such  data  for  any  Service.  Some  organizations  (e.g.,  Naval 
Sea  Systems  Command  and  the  Marine  Corps)  are  known  to  be  resistant  to  such 
measures.  The  additional  data  from  the  depots  will  not  necessarily  remove  resis¬ 
tance.  It  is  possible  that  the  increased  cost  of  gathering  more  detailed  data  will  not 
be  recouped  in  an  increased  analysis  capability. 

We  analyzed  and  summarized  general  overriding  trends  and  noted  specific  trends, 
when  appropriate.  We  stress  that  the  performance  metrics  should  provide  insight 
into  how  well  an  organization  is  functioning  but  should  not  drive  the  organiza¬ 
tion’s  functions.  There  is  always  the  danger  that  emphasis  will  be  placed  on  im¬ 
proving  the  “scores”  of  the  metrics,  and  that  attention  will  be  paid  to  improving 
the  components  of  the  metrics  at  the  expense  of  other  factors. 
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Chapter  1 

Introduction 


Overview 

Many  Department  of  Defense  (DoD)  support  organizations  are  financed  with  re¬ 
volving  funds  to  take  advantage  of  the  flexibility  offered  by  them.  However,  in 
the  appropriations-oriented  Defense  Programming  Review,  there  has  been  little 
visibility  of  the  internal  financial  operating  plans  of  those  support  organizations 
and  of  the  total  funding  that  their  customers  plan  to  spend  on  them. 

Additionally,  even  when  estimates  of  available  customer  funding  are  made,  cus¬ 
tomers  may  change  their  workload  requirements  for  a  variety  of  reasons,  includ¬ 
ing  changing  priorities  or  national  security  requirements.  The  cost  of  these 
changes  to  customer  requirements  is  not  well  understood.  Also,  the  lack  of  coor¬ 
dinated  plans  for  both  support  customers  and  support  providers  results  in  DoD’s 
inability  to  provide  effective  oversight  and  management  guidance  of  major  sup¬ 
port  organizations  financed  through  the  revolving  or  working  capital  funds. 

Objective 

DoD  is  interested  in  developing  performance  metrics  applicable  to  program 
analysis  of  defense  support  business  areas.  In  this  report,  we  identify  key  indica¬ 
tors  geared  toward  evaluating  resource  utilization  during  program  reviews  and  at 
other  times. 

This  is  the  first  in  a  series  of  reports  focusing  on  the  various  activities  in  the  De¬ 
fense  Working  Capital  Fund  (DWCF).  In  this  report,  we  highlight  the  Depot 
Maintenance  business  area.  Specifically,  our  research  and  findings  focus  on  three 
specific  performance  metric  areas  requested  by  the  sponsor.  These  areas  include 
output,  cost,  and  quality.  In  performing  this  research,  we  concentrated  on  identi¬ 
fying  potential  metrics  in  these  three  specific  areas  and  evaluating  the  existing 
supporting  data  available  and  its  validity.  We  also  observed  the  collected  data,  the 
metrics,  and  the  potential  indications  of  those  data. 

Related  Research 

Concurrent  with  our  Depot  Maintenance  business  area  research,  we  have  begun  a 
more  broad  review  of  the  Supply  Management  business  area  of  all  the  military 
services  and  the  Defense  Logistics  Agency  (DLA).  This  concurrent  effort  is  being 
conducted  with  fewer  research  constraints  in  the  early  phases.  This  may  then  al¬ 
low  more  potential  performance  metrics  to  be  considered,  at  least  initially,  prior  to 
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narrowing  the  focus  of  the  research  through  the  addition  of  constraints,  such  as 
immediate  data  availability. 

We  anticipate  that  in  the  subsequent  efforts,  due  to  fewer  constraints,  we  will  be 
able  to  place  more  emphasis  on  areas  such  as  the  following: 

♦  Are  the  measures  we  developed  linked  well  to  key  factors? 

♦  What  cautions  do  we  have  about  the  use  of  these  measures  (and  these 
data)  by  senior  analysts  and  managers? 

♦  Are  there  emerging  areas  or  practices  that  must  now  be  measured  that 
were  not  required,  or  even  known,  in  the  past?  If  so,  how  long  will  be  re¬ 
quired  to  develop  a  valid  baseline? 

We  anticipate  that  following  the  completion  of  the  Depot  Maintenance  and  Sup¬ 
ply  Management  business  area  research,  subsequent  research  will  be  desired  in 
other  activity  groups  of  the  DWCF.  These  areas  include,  at  a  minimum,  the  Dis¬ 
tribution  and  Transportation  business  areas. 

Constraints 

To  minimize  the  disruption  to  Office  of  the  Secretary  of  Defense  (OSD)  activities, 
agencies,  and  the  Services  during  the  course  of  this  task,  it  was  decided  that  new 
data  reporting  requirements  would  be  minimized.  Therefore,  it  was  necessary  for 
the  Logistics  Management  Institute  (LMI)  to  use  existing  data  elements  to  the 
maximum  extent  possible.  This  constraint  affects  both  historical  and  projected 
data. 
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Chapter  2 

Methodology 


Performance  Perspectives 

Our  approach  to  determining  appropriate  performance  measures  began  with  a  re¬ 
view  of  metrics  literature,  a  review  of  DoD  strategic  documents  emphasizing  ar¬ 
eas  focusing  on  logistics,  a  review  of  performance  measure  documents  submitted 
to  OSD,  and  a  matching  of  DoD  focus  areas  to  the  perspectives  highlighted  in  the 
literature.  Then  we  collected  available  data  and  developed  metrics  based  on  the 
basis  of  those  data. 

We  began  by  initiating  a  literature  review.  The  literature  review  provided  a  com¬ 
mon  theme  that  insisted  performance  measures  needed  to  be  linked  to  the  organi¬ 
zation’s  strategic  objectives  to  be  effective  tools  for  business  improvement.  The 
linkage  is  normally  achieved  by  identifying  performance  measures  that  highlight 
accomplishment  of  key  success  factors.  Such  factors  are  required  for  the  organi¬ 
zation  to  achieve  its  strategic  objectives. 

Performance  areas  are  measured  for  several  reasons,  such  as  for  planning  and 
budgeting,  assessing  operational  improvements,  allocating  resources,  and  moni¬ 
toring  operations.  According  to  Ammons,  there  are  four  categories  of  perform¬ 
ance  measures:1 

♦  Workload  measures — indicate  the  amount  of  work  performed  or  the 
amount  of  services  received  (e.g.,  number  of  items  repaired). 

♦  Efficiency  measures — relate  the  work  performed  to  the  resources  con¬ 
sumed  to  perform  the  work  (e.g.,  cost  per  repair). 

♦  Effectiveness  measures — indicate  the  quality  of  performance  by  as¬ 
sessing  the  level  of  performance  objectives  achieved  (e.g.,  percentage 
of  correct  repairs). 

♦  Productivity  measures — combine  efficiency  and  effectiveness  meas¬ 
ures  into  one  metric  (e.g.,  unit  cost  per  correct  repair). 

The  “balanced  scorecard”  is  a  popular  methodology  that  looks  across  functional 
areas  within  an  organization  (to  ensure  one  function  is  not  optimized  at  the  ex¬ 
pense  of  another)  and  ties  the  strategic  objectives  of  the  organization  directly  to 


1  David  N.  Ammons,  Municipal  Benchmarks:  Assessing  Local  Performance  and  Establishing 
Community  Standards.  Thousand  Oak,  CA:  Sage  Publications,  1996. 
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the  performance  measures.2  Figure  2-1  illustrates  such  a  strategic  measurement 
model  that  expands  upon  the  balanced  scorecard  approach. 

Figure  2-1.  Strategic  Measurement  Model 
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{ 

{ 
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•  What  the  organization  is. 

•  The  future  goal(s)  of  the  organization. 

•  What  the  organization  stands  for. 

•  What  the  organization  needs  to  focus  on 
to  beat  its  competitors  and  achieve  its 
vision. 


•  A  balanced  scorecard 
(past-present-future). 


•  The  desired  annual  and  long-term 
levels  for  each  metric. 


•  Activities  implemented  to  achieve  the  goals. 


The  first  three  steps  in  the  process  are  emphasized  in  Figure  2-1.  This  highlights 
the  areas  addressed  in  our  analysis.  The  goals,  objectives,  and  strategies  must  be 
developed  internally  by  the  organization’s  management  leaders  (at  many  levels). 

We  used  the  model  shown  in  Figure  2-1  as  a  guide  for  our  analysis.  First  we 
identified  and  examined  key  DoD  documents  for  the  mission  and  values  descrip¬ 
tions.  Although  the  mission  and  values  forming  the  foundation  for  a  measurement 
system  normally  are  developed  through  interaction  with  senior  leaders,  constraints 
of  this  analysis  required  that  a  review  of  the  key  documents  capture  this  informa¬ 
tion.  We  next  reviewed  key  Service  and  agency  logistics  plans  and  contracts  to 
identify  issues  and  areas  on  which  to  focus.  That  review  identified  the  key  success 
factors  required  to  attain  the  strategic  goals  of  the  logistics  community.  The  last 
step  was  to  review  other  relevant  data  sources,  as  summarized  in  Table  2-1,  to 
capture  measurements  already  reported  and  to  better  understand  the  data  already 
available.  The  relevant  data  sources  to  cover  each  of  the  first  three  steps  in  Figure 
2-1  are  summarized  in  Table  2-1. 


2  Robert  S.  Kaplan  and  David  P.  Norton,  “Putting  the  Balanced  Scorecard  to  Work,”  Harvard 

Business  Review,  September-October  1993,  pp.  134-149. 

3  Mark  Graham  Brown,  Keeping  Score— Using  the  Right  Metrics  to  Drive  World  Class  Perform¬ 

ance.  New  York:  Quality  Resources,  1996. 
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Methodology 


Table  2-1.  Measurement  Data  Sources 


Measurement  model  step 

Data  source 

Mission,  vision,  and  values 

•  DoD  Logistics  Strategic  Plan  (1 998) 

•  DLA  and  Service  strategic  logistics 
plans 

•  Defense  Reform  Initiative 

•  DLA  performance  contract  with  De¬ 
fense  Management  Council 

Key  success  factors  and  business  funda¬ 
mentals 

•  Defense  Science  Board  report  on  ac¬ 
quisition  reform  (March  1998) 

•  Recent  performance  updates  to  service 
logistics  leadership 

•  Headquarters  and  National  Inventory 
Control  Point  interviews 

•  Past  budget  submissions 

Performance  metrics 

•  Past  Program  Objective  Memorandum 
submissions 

•  DLA  performance  contract  with  De¬ 
fense  Management  Council 

•  LMI  research 

As  mentioned  earlier,  each  literature  source  used  some  type  of  categorization 
method  to  group  and  identify  the  minimum  areas  that  require  measurement,  such 
as  the  four  mentioned  earlier  (workload,  efficiency,  effectiveness,  and  productiv¬ 
ity).  If  businesses  and  organizations  are  viewed  as  transformational  processes 
(transforming  a  given  input  into  a  desired  output),  then  there  are  three  measure¬ 
ment  categories  that  relate  inputs  to  outputs: 

♦  Productivity  (actual  output/actual  input) 

♦  Utilization  (capacity  used/available  capacity) 

♦  Performance  (actual  output/standard  output). 

One  popular  method  of  identifying  these  metrics,  ensuring  they  measure  the  im¬ 
pact  on  the  organization  as  a  whole  (not  just  one  department),  and  tying  them 
back  to  the  organization’s  strategic  goals  is  the  Balanced  Scorecard.  Although  we 
do  not  implement  the  Balanced  Scorecard  methodology  in  this  analysis,  we  used 
its  principles  to  try  to  develop  metrics  that  were  meaningful  across  functional  ar¬ 
eas. 
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For  example,  one  must  look  at  performance  measures  from  many  viewpoints  as 
indicated  in  Table  2-2: 


Table  2-2.  Performance  Perspectives 


Point  of  view 

Question  to  ask  of  the  organization 

Performance  measures 

Financial 

How  do  shareholders  or  sponsors 
perceive  us? 

•  Budget  compliance 

•  Return  on  investment 

Customer 

How  do  customers  see  us? 

•  Customer  ranking  surveys 

•  Order  backlog 

Internal 

In  what  area  must  we  excel? 

•  Project  close-out  cycle 

•  Rework 

Growth/innovation 

Can  we  continue  to  improve  and 
create  value? 

•  Revenue  from  new 
sources 

In  addition  to  the  points  of  view  evidenced  in  Table  2-2,  one  might  also  measure 
employee  satisfaction  and  supplier  performance.  Although  not  traditionally  in¬ 
cluded  in  the  Balanced  Scorecard,  these  two  areas  have  been  receiving  attention 
in  recent  studies. 

Thus,  it  is  important  to  look  across  functional  areas  within  the  organization,  to 
ensure  one  department  is  not  measured  extensively  and  optimized  at  the  expense 
of  others.  Additionally,  it  is  important  to  establish  linkages  between  performance 
metrics  and  the  organization’s  strategic  goals. 

In  the  next  chapter,  we  discuss  the  specific  metrics  that  were  developed  and  asso¬ 
ciated  trends.  Appendix  A  contains  Service-specific  results  for  each  of  the  per¬ 
formance  areas. 

Performance  Metric  Areas 

The  measurement  literature  points  to  a  common  theme  that  recommends  perform¬ 
ance  measures  be  linked  to  the  organization’s  strategic  objectives. 

In  this  report,  we  concentrate  on  three  key  depot  maintenance  performance  areas: 
output,  cost,  and  quality.  These  areas  may  be  mapped  as  follows:  output  reflects 
workload  measures,  cost  is  used  as  an  efficiency  measure,  and  quality  corre¬ 
sponds  to  effectiveness  measures.  Other  categorization  methods  will  be  discussed; 
however,  in  each  case,  the  three  performance  areas  are  easily  mapped  to  the  rec¬ 
ommended  categorization. 

Outputs  are  defined  as  indicators  of  activity  levels.  In  the  depot  maintenance  envi¬ 
ronment,  such  outputs  include  labor  hours,  number  of  items  processed,  and  profit 
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streams  (for  commercial  maintenance  enterprises).4  Outputs  also  may  serve  as 
normalization  factors,  to  better  compare  between  activities  that  may  not  be  di¬ 
rectly  comparable. 

The  two  major  categories  of  cost  we  consider  are  operating  costs  and  capital 
costs.  Operating  costs  are  funds  expended  to  pay  for  the  day-to-day  operations  of 
the  facility,  such  as  labor,  materials,  and  utilities.  Operating  costs  may  provide 
indications  of  business  base  adequacy  by  identifying  movement  of  labor  hours 
from  direct  to  indirect  accounts,  indicating  lack  of  flexibility  in  labor  manage¬ 
ment,  for  example.  Distinctions  between  direct  and  indirect  costs  must  be  under¬ 
stood  to  properly  calculate  meaningful  indicators.  Capital  costs  are  funds 
expended  for  procuring  equipment,  facilities,  and  other  nonrecurring  expenses. 
They  can  be  used  to  analyze  the  amount  of  investment  an  activity  is  making.  This 
can  be  viewed  as  an  indicator  of  up-front  investment  to  minimize  future  operating 
costs,  both  labor  and  material.  In  this  analysis,  we  combine  this  performance  met¬ 
ric  with  others  to  determine  ratios  such  as  throughput.  Such  ratios  provide  insight 
into  the  use  that  capital  assets  are  receiving. 

Measuring  quality  with  quantitative  metrics  is  often  difficult.  In  this  analysis,  we 
investigated  two  aspects  of  quality:  customer  satisfaction  and  repair  cycle  time 
(timeliness  and  thoroughness  of  repairs).  Often,  the  data  are  not  directly  available 
to  measure  quality,  so  surrogate  data — with  its  inherent  shortcomings — must  be 
used  instead.  For  example,  data  measuring  repair  thoroughness  is  rarely  captured. 
However,  if  reliability  data,  such  as  mean  time  between  failure  data,  are  available, 
then  adequate  measures  of  quality  often  can  be  determined. 


4  Labor  hours  are  typically  an  input  measure,  but  we  used  them  as  a  surrogate  for  output.  See  our 
discussion  in  Chapter  3. 
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Chapter  3 

Analysis 


As  stated  earlier,  this  analysis  concentrates  on  three  key  depot  maintenance  per¬ 
formance  areas:  output,  cost,  and  quality.  Each  is  discussed  in  a  section  that  fol¬ 
lows.  Data  sources  and  results  associated  with  each  metric  also  are  explained. 


Output 


Direct  labor  hours  (DLH)  were  used  as  the  output  metric.  DLH  is  traditionally  an 
input  measure,  but  it  can  also  be  used  to  assess  output,  since  DLH  can  be  used  as 
a  surrogate  for  measurement  of  the  “standard  item  repaired,”  the  ideal  output 
measure.  To  compute  standard  item  repaired,  detailed  data  on  the  number  of  indi¬ 
vidual  items  of  equipment  repaired  must  be  collected.  Additionally,  several  stan¬ 
dard  hours  to  make  each  type  of  repair  to  each  type  of  equipment  must  be 
available.  With  this  information,  hours  and  items  can  be  interchanged  with  a  sim¬ 
ple  conversion.  These  data  were  unavailable;  therefore,  DLH  was  selected  as  the 
surrogate.  Additionally,  DLH  is  an  item-independent  metric.  It  measures  total 
hours  consumed  and  does  not  discriminate  between  the  types  of  items  repaired. 
Finally,  DLH  highlights  activity-level  changes  that  can  be  used  to  normalize  other 
measures.  This  point  will  become  more  evident  as  subsequent  metrics  are  exam¬ 
ined. 

In  a  constant  workload  environment,  the  standard  objective  is  to  reduce  DLH. 

This  often  is  achieved  by  capital  investments  in  equipment  to  reduce  labor  hours 
or  process  improvements  to  improve  efficiency.  We  reemphasize  that  any  metric, 
when  assessed  in  isolation,  may  promote  illogical  behavior,  and  DLH  is  no  ex¬ 
ception.  Actions  to  drive  DLH  down  must  be  balanced  with  the  “reactions”  in 
other  areas.  These  other  areas  will  be  discussed  later  in  this  report. 

Sometimes,  an  increasing  DLH  is  acceptable,  for  example,  when  hours  move 
from  overhead  to  DLH.  This  causes  DLH  to  increase  but  decreases  the  ratio  of 
operating  expense  to  DLH. 

Data 


The  data  sources  for  this  metric  (and  all  other  metrics  discussed  in  this  report) 
were  the  Depot  Maintenance  Operations  Indicators  (DMOI)  and  the  Budget  Esti¬ 
mate  Submission  (BES).  The  DMOI  reports  direct  labor  hours  expended  for  item 
repair,  by  depot.  These  historical  data  were  summarized  to  the  service  level  for 
the  analysis.  The  BES  data  are  projections  of  DLH  for  the  budget  years  and  were 
extracted  from  the  FY99  BES.  DMOI  data  were  used  to  display  metrics  for  the 
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period  FY93  to  FY96.  BES  data  then  were  used  to  show  DLH  projections  from 
FY97  to  FY99. 

A  problem  with  BES  data  is  that  not  all  Services  report  military  labor  hours  asso¬ 
ciated  with  depot  operations.  For  example,  the  Army  included  military  manpower 
in  direct  labor  totals  while  the  Air  Force  DLH  aggregates  were  not  found  in  the 
BES.  Therefore,  only  civilian  direct  labor  was  used  in  the  analysis  and  was  com¬ 
puted  from  BES  detailed  displays.  The  data  for  the  services  that  explicitly  re¬ 
ported  military  labor  hours  highlighted  that  the  military  hours  are  dwarfed  by  the 
civilian  labor  hours  and  that  the  error  associated  with  this  omission  would  be 
small. 

An  additional  problem  was  that  the  Navy  did  not  consistently  report  closing  ship¬ 
yards’  and  depots’  hours  in  FY97.  The  BES  data  did  not  allow  this  to  be  corrected 
and,  therefore,  is  a  factor  that  must  be  considered  when  reviewing  the  metrics. 


Results 


The  DMOI  and  BES  data  for  DLH  were  graphed  for  each  Service.  The  Navy  data 
were  reported  for  both  Naval  Air  Systems  Command  (NAVAIR)  and  Naval  Sea 
Systems  Command  (NAVSEA),  so  each  was  graphed  separately.  Figure  3-1 
shows  the  DLH  for  each  Service. 

Figure  3-1.  Direct  Labor  Hours 


Note:  USAF  =  U.S.  Air  Force;  USMC  =  U.S.  Marine  Corps. 

Because  each  Service  has  different  workload  levels,  different  accounting  systems, 
and  different  reporting  requirements,  the  data  displayed  in  Figure  3-1  (and  all 
other  graphs  in  this  report)  are  not  directly  comparable  between  the  Services.  Our 
goal  is  to  explore  the  relative  trends  that  are  exhibited  and  note  where  the  Services 
may  either  show  similar  patterns,  or  where  substantial  divergence  is  evident. 
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Analysis 


Figure  3-1  shows  the  DLH  for  the  Air  Force  and  NAVSEA  are  showing  a  down¬ 
ward  trend,  the  Army  and  NAVAIR  are  slightly  increasing,  and  the  Marine  Corps 
DLH  has  remained  relatively  flat,  with  a  slight  decrease. 

For  the  Army,  a  substantial  (approximately  25  percent)  decrease  in  workload  oc¬ 
curred  early  in  the  reported  period,  from  FY93  to  FY94.  This  is  not  unexpected 
due  to  the  significant  drawdowns  in  the  early  1990s.  A  smooth  transition  does  not 
take  place  during  the  transition  from  DMOI  to  BES  data.  The  Army’s  subsequent 
rise  in  DLH  is  due  to  BES  data  that  include  munitions  depot  maintenance,  which 
is  not  included  in  the  DMOI  data.  Since  the  BES  data  are  not  reported  at  the  depot 
level,  specific  depots’  data  cannot  be  removed.  In  summary,  the  Army  DLH  trend 
is  not  an  area  of  concern. 

The  Air  Force’s  DLH  decreased  at  a  relatively  constant  rate  (approximately  5  per¬ 
cent  per  year).  The  transition  point  from  the  DMOI  data  to  the  BES  data  appears 
to  be  smooth,  which  is  the  desired  behavior.  Unlike  the  Army  data,  it  appears  that 
the  Air  Force  DMOI  and  BES  data  are  directly  comparable.  Additionally,  the 
downward  trend  is  a  first-level  indicator  of  actions  being  taken  to  reduce  DLH. 

NAVAIR  data  indicated  that  workload  increased  in  the  FY94  to  FY96  time  frame 
and  is  predicted  to  remain  constant  through  FY99.  The  transition  from  DMOI  to 
BES  data  is  smooth. 

The  NAVSEA  output  data  indicate  that  workload  is  decreasing  at  a  consistent  rate 
over  the  time  frame  plotted,  with  the  transition  between  DMOI  and  BES  data  be¬ 
ing  seamless. 

The  USMC  data  indicate  that  workload  is  decreasing  at  a  slow,  yet  constant  rate, 
with  a  somewhat  larger  drop  planned  for  FY99.  Again,  the  transition  from  DMOI 
to  BES  is  also  smooth. 


Operating  Cost 


Two  metrics  were  developed  for  operating  cost.  The  first  is  the  sum  of  direct  la¬ 
bor  costs  (DLCs)  and  overhead  costs  (we  define  these  two  components  together  as 
the  operating  expense),  divided  by  (DLH),  as  shown  in  Equation  3-1. 


{Direct  labor  cost  +  Overhead  cost) 
Direct  labor  hours 


=  Metric  #1 . 


[Eq.  3-1] 


This  metric  reflects  total  cost,  excluding  direct  material,  per  DLH.  The  focus  is  on 
costs  controllable  by  the  depot.  Material  costs  are  a  pass-through  cost  not  con¬ 
trolled  by  the  depot,  and  therefore  are  not  included. 
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The  second  metric  was  overhead  cost  divided  by  DLH  (Equation  3-2): 

Overhead  cost 


Direct  labor  hours 


=  Metric  #2 . 


[Eq.  3-2] 


The  combination  of  these  two  metrics  captures  the  total  nonmaterial  cost  changes. 
Therefore,  these  metrics  isolate  the  changing  elements  that  are  controlled  by  the 
depots.  Additionally,  when  these  metrics  are  studied  in  combination  with  the 
DLH  output  metric,  one  can  get  an  indication  of  business  base  requirements.  For 
example,  if  Equation  3-1  shows  an  increasing  trend  and  DLH  has  remained  con¬ 
stant,  then  costs  are  increasing.  If,  simultaneously.  Equation  3-2  shows  a  de¬ 
creasing  trend,  then  the  combination  of  these  three  metrics  that  includes  DLH, 
isolates  direct  labor  costs  as  the  increasing  component. 

The  desired  trend  is  for  operating  expenses  to  increase  at  a  slower  rate  than 
throughput  (total  gross  sales  less  materials)  in  an  increasing-workload  environ¬ 
ment.  That  means  in  a  constant-workload  environment,  the  goal  is  for  operating 
expenses  to  decrease.  Such  performance  results  from  reducing  the  overhead  com¬ 
ponents  of  labor  and  materials  and  reducing  direct  labor  through  efficiency  in¬ 
creases.  A  danger  lies  in  focusing  on  this  metric  and  reducing  DLH  alone.  The 
result  is  that  the  ratio  of  operating  expense  to  DLH  will  increase  and  an  incorrect 
conclusion  could  be  drawn.  Thus,  we  emphasize  that  metrics  and  trends  need  to 
be  evaluated  as  a  set  of  information  so  that  the  correct  conclusions  can  be  drawn. 


Data 

The  DMOI  reports  a  single  line  item  for  operating  expenses,  which  arc  total  costs 
less  direct  materials.  Therefore,  operating  expenses  are  composed  of  direct  labor, 
indirect  labor,  and  indirect  materials.  That  means  that  the  numerator  in  Equation 
3-1,  although  described  as  the  sum  of  two  components,  was  captured  by  the  oper¬ 
ating  expense  line  item  of  the  DMOI.  However,  not  until  FY96  was  overhead  re¬ 
ported  directly  in  the  DMOI.  Since  the  DMOI  does  not  itemize  the  overhead 
components  separately,  overhead  could  not  be  computed  for  years  prior  to  FY96. 
Therefore,  Equation  3-2  could  not  be  computed  prior  to  FY96.  Again,  all  cost  in 
this  report  are  presented  in  current  year  dollars. 

The  BES  reports  direct  labor,  indirect  labor,  and  general  and  administrative 
(G&A)  labor  separately,  so  the  numerator  of  Equation  3-1  was  computed  when 
BES  data  were  used.  However,  indirect  materials  were  not  included  in  BES  data, 
but  they  were  in  the  DMOI  values.  Therefore,  when  comparing  the  numerator  for 
Equation  3-1,  we  must  realize  that  the  BES  and  DMOI  data  are  not  directly  com¬ 
parable. 

As  noted  earlier,  the  BES  did  not  report  overhead  as  a  separate  line  item,  so  it  was 
computed  for  Equation  3-2  as  the  sum  of  indirect  and  G&A  labor.  Again  the  BES 
data  do  capture  indirect  materials  while  the  DMOI  overhead  value  does  not  cap¬ 
ture  this  element. 
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Analysis 


Results 


The  two  operating  cost  metrics  are  graphed  in  Figure  3-2  and  Figure  3-3  for  each 
Service.  For  the  first  metric  ([ Direct  labor  cost  +  Overhead  cost]/DLH),  Figure 
3-2  illustrates  that  the  DMOI  data  exhibited  much  wider  swings  in  magnitude, 
while  the  BES  data  are  relatively  flat  and  of  similar  magnitude  for  each  Service. 
This  may  be  an  indication  that  the  indirect  material  costs  included  in  the  DMOI 
values  have  a  wider  variance  than  the  direct  and  overhead  costs.  However,  in  each 
case,  the  BES  data  are  less  than  the  DMOI  data,  as  would  be  expected. 

Figure  3-2.  Operating  Expense/DLH 


The  NAVAIR  FY95  datapoint  departs  significantly  from  an  otherwise  consistent 
FY93  to  FY96  trend.  Further  investigation  found  that  the  operating  expense  re¬ 
ported  in  the  third  quarter  of  FY95  was  negative  for  all  three  NAVAIR  entities. 
The  data  will  not  be  changed;  however,  this  data  point  should  be  ignored. 

Next,  we  need  to  look  at  the  data  for  the  second  metric  (see  Figure  3-3),  so  that 
comparisons  between  the  two  metrics  can  be  made  and  meaningful  assertions  will 
be  possible.  Figure  3-3  shows  the  results  of  graphing  Overhead  cost/DLH  for  each 
Service.  (As  noted  above,  data  were  only  available  beginning  in  FY96.)  One 
overwhelming  trend  appears  to  be  the  decrease  in  value,  and  then  a  general  level¬ 
ing  off,  for  all  Services  except  the  Air  Force,  which  shows  and  increase  and  then  a 
leveling  off.  However,  we  refrain  from  drawing  any  conclusions  at  this  time, 
since  we  only  have  1  year  of  reported  DMOI  data,  compared  with  3  years  of  com¬ 
puted  BES  values.  A  partial  explanation  of  the  large  drop  in  {Direct  labor  cost  + 
Overhead  cost)/DLH  from  DMOI  to  BES  data  may  be  that  the  DMOI  values  for 
overhead  include  indirect  material  costs,  while  the  BES  data  do  not.  The  relatively 
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flat  values  in  the  BES  period  may  indicate,  as  similarly  discussed  under  Figure 
3-2  that  the  indirect  material  costs  included  in  the  DMOI  values  have  a  wider 
variance  than  the  overhead  costs. 

Figure  3-3.  Overhead  Cost/DLH 


Fiscal  year 


Now  we  will  discuss  Service-specific  observations,  for  both  Figure  3-2  and  Figure 
3-3. 

For  the  Army,  the  change  of  the  DLH  baseline  (the  BES  data  include  munitions 
depot  maintenance,  but  DMOI  data  do  not)  has  caused  the  BES  data  to  be  gener¬ 
ally  lower  than  the  DMOI  data.  The  Operating  expense/DLH  and  Overhead 
cost/DLH  indicate  that  the  Army  depots  do  well  in  an  increasing-workload  envi¬ 
ronment  (FY97-FY98).  When  DLH  increases,  if  both  overhead  and  DLC  increase 
proportional  to  DLH,  the  Operating  expense/DLH  curve  would  be  flat.  However, 
since  the  curve  slopes  downward,  operating  expenses  are  increasing  at  a  slower 
rate  than  DLH.  Similarly,  the  downward  sloping  Overhead  cost/DLH  line  indi¬ 
cates  that  overhead  is  also  increasing  at  a  slower  rate  than  DLH. 

Trends  in  decreasing  workload  for  the  Army  depot  are  not  as  favorable.  Operat¬ 
ing  expense/DLH  does  not  increase  dramatically  in  FY98-FY99  when  DLH  in¬ 
creases.  This  indicates  that  operating  expenses  are  reduced  at  about  the  same  rate 
as  the  workload  decrease.  The  slight  growth  shown  by  the  Overhead  cost/DLH 
line,  in  combination  with  the  almost  flat  line  for  Operating  expense/DLH  indi¬ 
cates  that  overhead  is  decreasing  less  rapidly  than  DLH.  In  fact,  these  curves  indi- 
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cate  that  DLC  is  potentially  decreasing  faster  than  DLH,  and  overhead  is  becom¬ 
ing  the  more  substantial  component,  which  is  not  a  desired  outcome. 

Overhead,  as  an  individual  data  element,  was  reported  only  for  a  subset  of  the  Air 
Force  depots  in  the  DMOI.  Therefore,  the  FY96  Overhead  cost/DLH  data  point  is 
not  valid.  Focusing  on  the  FY97-FY99  time  frame,  DLH  decreases  in  both  peri¬ 
ods.  For  the  period  FY97-FY98,  Overhead  cost/DLH  and  Operating  ex- 
pense/DLH  rise,  indicating  that  at  least  overhead  is  not  decreasing  at  the  rate  of 
DLH.  The  FY98-FY99  line  segments  indicate  that  overhead  begins  to  decrease 
more  rapidly  than  DLH,  but  DLC  does  not  decrease  as  fast  as  DLH.  The  combi¬ 
nation  of  these  two  effects  indicates  that  DLC  does  not  decrease  at  a  rate  propor¬ 
tional  to  DLH. 

For  NAVAIR,  if  the  FY95  and  FY96  data  were  averaged,  the  trend  for  operating 
expenses  and  Operating  expense/DLH  would  be  a  decreasing  function — that  av¬ 
erage  data  point  would  be  the  same  as  the  FY97  BES  computed  point.  Because 
Operating  expense/DLH  and  Overhead  cost/DLH  remain  constant  or  decrease 
while  DLH  increases,  NAVAIR  is  doing  well  controlling  its  expenses. 

When  DLH  either  increases  or  decreases,  one  would  desire  both  Operating  ex¬ 
pense/DLH  and  Overhead  cost/DLH  to  decrease.  This  is  not  the  case  with  the 
NAVSEA  data.  During  FY97-FY98,  Overhead  cost/DLH  is  constant  but  Oper¬ 
ating  expense/DLH  increases.  This  indicates  that  DLCs  are  not  decreasing  at  the 
same  rate  as  DLH,  which  means  that  there  may  be  a  problem  with  DLC.  During 
FY98-FY99,  Overhead  cost/DLH  increases,  indicating  that  overhead  is  not  de¬ 
creasing  at  the  same  rate  as  DLH,  which  is  a  potential  problem.  This  may  be  a 
case  where  the  workload  may  be  constant  but  labor  hours  are  just  being  reclassi¬ 
fied  as  overhead  from  the  direct  category. 

For  the  Marine  Corps,  we  see  that  the  more  substantial  drop  in  FY98-FY99  DLH 
did  not  have  a  corresponding  decrease  in  overhead.  It  appears  that  DLC  decreased 
(at  least  proportionally  to  DLH),  but  overhead  did  not  decrease  proportionately. 

Capital  Cost 


Two  metrics  were  developed  for  capital  cost.  The  first  is  the  Capital  effectiveness 
ratio,  defined  as  throughput  divided  by  depreciated  capital  asset  value.  For  our 
purposes,  throughput  is  calculated  as  revenue  less  direct  material  cost,  as  shown 
in  Equation  3-3. 


Revenue  -  Direct  material  cost  ..  .  „„ 

- - —  =  Metric  #3 . 

Depreciated  capital  asset  value 


[Eq.  3-3] 


The  metric  described  by  Equation  3-3  represents  the  “value  added”  by  the  depot, 
per  unit  of  asset  value.  For  example,  the  increase  in  value  of  a  repaired  item  over 
a  damaged  item  is  due  to  two  components:  the  repairs  added  and  the  labor  re¬ 
quired  to  install  these  items.  The  depot  does  not  control  the  cost  of  materials  (a 
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pass-through  cost  to  the  depot)  but  does  control  the  labor  component.  This  is  the 
value  added  being  captured  in  this  measure. 

The  second  metric  is  simply  the  measure  of  Net  capital  investment  during  the  pe¬ 
riod.  The  DMOI  reports  this  value  as  “capital  assets,”  so  for  DMOI  data,  the  in¬ 
cremental  year-to-year  change  was  used.  That  is,  DMOI  reports  total  capital  each 
period,  so  the  net  capital  inventory  was  calculated  as  DMOh+i-DMOf.  The  BES 
did  not  report  this  metric  directly,  so  the  net  capital  investment  value  had  to  be 
calculated  with  given  BES  data,  as  shown  in  Equation  3-4: 

Program  year  authority  -  Depreciation  =  Metric  #4 .  [Eq.  3-4] 

Net  capital  investment  represents  the  degree  of  modernization  and  the  resources 
put  toward  the  infrastructure  required  to  support  the  mission.  This  second  metric, 
Net  capital  investment,  is  used  to  allow  for  the  separation  of  capital-only  increases 
from  throughput-only  decreases. 

Capital  investments  are  not  always  intended  to  increase  throughput,  but  may  be 
made  to  decrease  operating  expenses.  Increasing  investment  causes  the  ratio  of 
throughput  to  capital  to  decrease — however,  there  is  no  impact  on  revenue  (the 
customer  does  not  pay  a  different  price),  and  there  is  no  direct  material  change. 
Instead,  operating  expenses  and  overhead  may  decrease  in  relationship  to  DLH. 
Investments  can  be  effectively  used  to  decrease  labor  requirements.  Again,  this 
emphasizes  the  need  to  examine  all  metrics,  not  one  metric  in  isolation. 

Data 


The  Capital  effectiveness  ratio  is  reported  directly  in  the  DMOI.  The  DMOI  no¬ 
menclature  is  “capital  investment  effectiveness.”  Additionally,  depreciated  assets 
are  reported  directly  in  the  DMOI.  However,  we  calculate  the  ratio,  since 
throughput  (the  numerator  of  Equation  3-3)  and  depreciated  capital  asset  value  is 
directly  reported  in  the  DMOI.  As  stated  earlier,  all  financial  data  in  this  report 
are  presented  in  current  year  dollars. 

The  BES  reports  capital  investment  and  depreciation  separately  for  each  period. 
Only  the  incremental,  not  the  cumulative,  investment  is  reported  so  Equation  3-4 
is  used  to  calculate  the  Net  capital  investment.  Subsequently,  Equation  3-4  also 
serves  as  the  denominator  for  Equation  3-3  when  BES  data  are  used.  Additionally, 
only  equipment  values  and  real  property  maintenance  (RPM)  investment  are  rep¬ 
resented  in  the  BES — military  construction  (MILCON)  is  not.  For  DoD,  equip¬ 
ment  and  RPM  capture  approximately  70  percent  of  capital  investments.  Also,  the 
only  equipment  investments  reported  in  working  capital  fund  (WCF)  budgets  are 
for  replacement  items;  new  equipment  requirements  are  funded  by  the  customer 
or  system  program  office  imposing  the  requirements.  Therefore,  BES  data  may  be 
under-reporting  capital  investments,  compared  with  the  DMOI  data.  These  differ¬ 
ences  may  help  us  to  understand  some  of  the  results  below. 

A  key  reason  for  establishing  performance  measurements  is  to  develop  insight 
into  specific  business  areas  and  to  enact  control  on  components  of  the  measures 
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when  they  are  seen  to  be  exceeding  acceptable  limits.  However,  for  depot  mainte¬ 
nance,  new  mission  requirements  (buildings  and  equipment)  are  not  elements  un¬ 
der  depot  control  and,  therefore,  depot  management  should  not  be  judged  on  these 
areas.  MILCON  is  a  separate  appropriation  and  is  not  part  of  WCF  BES.  Equip¬ 
ment  for  new  missions  is  part  of  the  equipment  systems  funding  stream  (also  not  a 
part  of  WCF  BES).1 

Only  replacement  and  repair  requirements  (for  buildings  and  equipment)  are  cost 
elements  that  a  depot  can  control.  The  BES  capital  investments  data  capture 
equipment  replacement  procurement  and  RPM. 

Therefore,  capturing  only  replacement  equipment  and  RPM  encompasses  only 
about  70  percent  of  the  investment.  However,  these  items  are  essentially  100  per¬ 
cent  of  the  investment  controlled  by  the  depots.  The  capital  investment  baseline 
will  not  affect  the  direction  of  the  trends  shown  in  the  graphs — only  the  slope 
(rate  of  change)  of  the  lines. 

Results 


The  two  capital  cost  metrics  are  graphed  below  for  each  Service.  Neither  the 
Capital  effectiveness  ratio  (Figure  3-4)  nor  the  Net  capital  investment  (Figure  3-5) 
metrics  exhibited  any  overriding  trends.  One  observation  is  that  the  BES  data  for 
capital  investment  in  Figure  3-5  appear  to  be  “tighter”  with  less  variance  than  the 
DMOI  data.  Service-specific  results  are  presented  in  the  paragraphs  that  follow. 

Figure  3-4.  Capital  Effectiveness  Ratio 


The  Army’s  throughput  (revenue  less  direct  materials)  increased  from  the  DMOI 
data  to  the  BES  data  because  ordnance  depots  were  included  in  the  BES,  but  not 
the  DMOI.  From  FY97  to  FY99,  Net  capital  investment  decreased.  With  this  in 
mind,  the  data  indicate  that  while  not  replenishing  the  capital  replacement  fund 
(i.e.,  net  investment  decreased),  throughput  rose.  We  cannot  isolate  the  contribu- 


1  New  equipment  is  about  25  percent  of  total  equipment. 
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tion  to  throughput  that  is  new  mission  or  new  equipment  that  is  “free  issue”  to  the 
depot;  however,  these  data  could  inspire  one  to  investigate  this  further. 


Figure  3-5.  Net  Capital  Investment 


Air  Force  throughput  increased  significantly  from  DMOI  to  BES  data  points.  We 
verified  our  computations  but  could  not  uncover  a  clear  reason  for  this  jump.  All 
that  can  be  said  is  that  the  trend  is  an  increase  of  the  capital  effectiveness  ratio, 
even  in  years  of  negative  net  investment.  This  indicates  the  maintainability  or  im¬ 
provement  of  throughput  with  a  negative  “capital  flow,”  at  least  temporarily  in  the 
time  frame  studied. 

NAV AIR’s  capital  cost  performance  was  “middle  of  the  road.”  One  observation  is 
that  throughput  tended  to  increase  with  low  net  investment.  However,  in  later 
years,  throughput  did  not  increase  substantially  by  the  increased  positive  net  in¬ 
vestment. 

NAVSEA  did  not  show  promising  capital  cost  performance.  Generally,  through¬ 
put  decreased  while  capital  investment  was  consistently  negative.  The  resulting 
indication  is  that  throughput  decreased  at  a  greater  rate  than  capital  assets. 

The  Marine  Corps’  performance  also  was  not  encouraging.  The  most  recent  data 
indicate  a  decreasing  trend  in  throughput,  while  capital  investment  hovered 
around  zero.  The  observation  is  that  throughput  decreased  at  a  greater  rate  than 
assets. 
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Analysis 


Quality 

Two  metrics  were  developed  to  measure  quality.  The  first  is  schedule  confor¬ 
mance,  defined  as  the  number  of  items  completed  on  time,  divided  by  the  number 
of  items  scheduled.  Equation  3-5  shows  this  relationship. 

Number  of  items  completed  on  time  ,,  .  r_  n 

- - - - -  =  Metric  #5 .  [Eq.  3-5] 

Number  of  items  scheduled 

Equation  3-5  captures  customer  satisfaction  quality,  since  not  meeting  schedule 
results  in  a  ripple  effect  to  depot  customers,  present  and  future.  Since  this  metric 
is  a  percentage,  only  an  increase  in  its  value  is  an  acceptable  change. 

A  major  shortfall  with  this  metric  is  that  schedule  conformance  weights  each 
work  order  equally,  whether  that  work  order  is  for  a  single  item  or  a  large  “lot”  of 
items.  However,  more  detailed  data  would  require  a  significant  data  collection 
effort. 

The  second  metric  is  in-repair  time,  defined  as  the  total  item-days  in  repair,  di¬ 
vided  by  the  number  of  items  completed.  This  metric  also  is  described  as  DMOI 
process  days. 

We  do  not  recommend  using  this  second  metric,  since  the  product  mix  can  cause 
a  significant  change  in  the  metric  value.  For  example,  one  would  be  adding  dispa¬ 
rate  items  such  as  tanks  and  rifles  without  any  type  of  normalization  process. 
Changing  a  depot’s  workload  from  tank  repair  to  small  arms  repair  would  cause  a 
significant  change  in  the  metric  value;  however,  the  mix  of  the  systems  is  not 
within  the  depot’s  control. 


Data 


The  DMOI  reported  “schedule  indicator”  data  were  used  directly  for  schedule 
conformance.  However  there  are  concerns  with  some  of  the  Services.  NAVSEA 
does  not  report  schedule  indicator  data  because  NAVSEA  defines  units  completed 
as  ships,  the  largest  depot-level  repairable.  Since  the  number  of  units  (ships)  com¬ 
pleted  in  a  given  time  frame  is  so  low  compared  with  standard  product  lines,  this 
indicator  would  give  misleading  or  inaccurate  results. 

The  Marine  Corps  also  does  not  report  the  schedule  indicator  since  it  does  not 
have  the  maintenance  float  that  allows  the  Fleet  Marine  Forces  (FMF)  to  issue 
weapon  systems  and  receive  replacements  to  satisfy  Table  of  Equipment  (TE) 
shortages.  Therefore,  in  most  cases,  the  FMF  operates  TE  deficient  until  depot 
maintenance  is  completed  and  the  weapon  is  returned.  The  Army  did  not  report 
this  data  in  FY93  and  FY94. 

The  BES  does  not  contain  data  that  are  comparable,  so  no  projections  were  made. 
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Results 


Since  there  were  only  two  data  points  for  the  Army,  we  can  only  review  the  Air 
Force  and  NAVAIR  data  for  trends  shown  in  Figure  3-6.  These  are  discussed  in 
the  Service-specific  paragraphs  below. 

Figure  3-6.  Schedule  Conformance 


< - DMOI - ► 


The  Army’s  average  schedule  conformance  increased  from  FY95  to  FY96;  how¬ 
ever,  with  only  two  data  points  available,  it  would  be  premature  to  call  this  a 
trend. 

The  Air  Force  schedule  conformance  appears  to  have  increased.  However,  noting 
the  scale  of  the  graph,  the  change  is  approximately  5  percent,  so  quality  may  be 
considered  constant  prior  to  and  following  the  jump  in  the  FY94-FY95  time 
frame. 

NAVAIR’ s  schedule  conformance  decreased  from  FY93  to  FY96  by  approxi¬ 
mately  13  percent.  This  metric  indicates  an  undesirable  trend  and  highlights  an 
area  for  improvement,  based  on  the  available  historical  data. 

As  discussed  above,  NAVSEA  and  the  Marine  Corps  do  not  report  schedule  data, 
since  their  end  items  and  logistics  systems  cause  such  data  to  be  misrepresentative 
of  true  depot  performance. 
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Chapter  4 

Findings,  Conclusions,  and  Cautions 


Our  findings  and  conclusions  pertain  to  the  two  broad  purposes  of  this  task:  (1) 
identifying  potential  metrics  and  evaluating  the  supporting  data;  and  (2)  observing 
the  collected  data,  the  associated  metrics,  and  the  potential  indications  of  the  met¬ 
rics.  This  chapter  also  includes  some  important  cautions  on  the  use  of  the  result¬ 
ing  metrics. 

Potential  Metrics  and  Supporting  Data 

We  believe  that  the  set  of  metrics  areas  (output,  cost,  and  quality)  presented  in 
this  report  can  be  useful  in  conducting  program  analysis  of  the  Depot  Mainte¬ 
nance  business  area.  The  metrics  can  be  linked  with  major  success  factors  associ¬ 
ated  with  the  Department’s  strategic  goals  and  missions.  These  metrics,  taken 
together,  form  a  reasonably  good  picture  of  resource-related  trends  in  output  and 
cost  performance. 

Sufficient  data  to  construct  the  output  and  cost  metrics  is  available  in  the  BESs. 
Because  of  changes,  to  include  transfers  of  missions  between  organizations,  care 
must  be  exercised  when  extracting  data  to  ensure  a  consistent  baseline  is  being 
used.  In  some  cases,  the  baseline  will  change  and  the  detailed  information  to  cor¬ 
rect  the  data  may  not  be  available.  At  this  point,  one  must  make  it  clear  to  the  us¬ 
ers  of  the  data  so  invalid  conclusions  will  not  be  drawn.  We  conclude  that 
analysis  of  output  and  cost  can  be  done  without  additional  data. 

We  found  quality  performance  data  to  be  a  substantial  shortfall.  Not  all  Services 
adequately  reported  quality  data  in  the  DMOI,  and  the  BES  does  not  include  such 
data  for  any  Service.  Some  organizations  (i.e.,  NAVSEA  and  the  Marine  Corps) 
are  known  to  be  resistant  to  such  measures.  The  additional  data  from  the  depots 
will  not  necessarily  remove  resistance.  It  is  possible  that  the  increased  cost  of 
gathering  more  detailed  data  will  not  be  recouped  in  an  increased  analysis  capa¬ 
bility. 
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General  Observations  About  the  Depot 
Maintenance  Business  Area 

We  analyzed  general  overriding  trends  and  noted  specific  trends,  when  appropri¬ 
ate.  A  summary  of  the  findings  is  as  follows: 

♦  Generally,  we  saw  that  all  activities  except  NAVAIR  were  experiencing 
decreasing  workload.  The  Army’s  increase  was  caused  by  more  depots  re¬ 
porting  in  the  BES  data  than  in  DMOI. 

♦  The  Army  exhibited  positive  capital-effectiveness  trends.  The  Army  had 
more  difficulty  balancing  direct  and  indirect  costs  in  a  decreasing- 
workload  environment  than  in  an  increasing-workload  environment. 

♦  The  Air  Force  showed  overhead  cost  decreases  that  were  more  than  pro¬ 
portional  to  DLH  decreases,  when  DLH  dropped  substantially.  Also,  the 
Air  Force’s  direct  labor  costs  did  not  appear  to  decrease  at  a  rate  consis¬ 
tent  with  DLH  changes. 

.  ♦  NAVAIR’ s  direct  labor  costs  and  overhead  costs  both  increased  at  a  rate 
no  greater  than  DLH  increases.  However,  DLH  did  increase,  especially  in 
the  FY94  to  FY96  time  frame. 

♦  NAVSEA  experienced  consistently  decreasing  DLH  over  the  entire  time 
frame  analyzed.  Overall,  NAVSEA’ s  capital  effectiveness  decreased,  but 
its  net  investment  was  negative.  This  begs  the  question  of  whether  the  de¬ 
crease  might  have  been  caused  by  the  lack  of  investment. 

♦  The  Marine  Corps  had  a  decreasing  workload,  most  significantly  in  the 
FY98  to  FY99  time  frame.  Capital  effectiveness  decreased  overall,  during 
a  period  with  relatively  no  net  capital  investment. 

We  stress  that  comparisons  among  these  organizations  are  essentially  invalid, 
since  the  data  sources,  reporting  requirements,  and  other  variables  differ,  resulting 
in  data  values  that  cannot  be  compared  directly.  The  trends  (i.e.,  increasing  or  de¬ 
creasing  metric  levels)  are  the  only  basis  for  comparison. 

Cautions 

We  offer  the  following  cautions: 

♦  The  three  depot  maintenance  performance  areas  are  part  of  a  larger  depot 
maintenance  system;  that  system  itself  is  part  of  an  even  larger  system. 

Like  all  complicated  systems,  it  is  difficult  to  analyze  subsets  in  isolation. 
Interactions  with  other  systems  are  important.  For  depot  maintenance,  these 
interactions  include  supply  management  and  transportation. 
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Findings,  Conclusions,  and  Cautions 


♦  We  stress  that  the  performance  metrics  should  provide  insight  into  how 
well  an  organization  is  functioning,  but  should  not  drive  the  organization’s 
functions.  There  is  always  the  danger  that  emphasis  will  be  placed  on 
achieving  an  improved  score  in  a  measured  area.  In  this  scenario,  attention 
will  be  paid  to  improving  the  performance  on  components  of  the  metrics, 
at  the  expense  of  other  factors. 

♦  Only  items  under  the  control  of  the  organization  should  be  measured. 
Measuring  things  outside  the  control  of  management  is  futile  and  dis¬ 
tracting,  since  management  cannot  necessarily  change  or  improve  the 
drivers  of  the  metric. 

♦  Metrics  should  not  be  static,  but  should  be  continuously  reviewed  to  en¬ 
sure  relevant  inputs  and  outputs  are  measured  and  that  the  metrics  provide 
information  that  is  useful. 

♦  No  single  metric  should  be  presented  and  evaluated  in  isolation.  This 
“family”  of  metrics  needs  to  be  considered  in  combination,  so  that  context 
and  interrelationships  can  be  fully  understood. 
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Appendix  A 

Detailed  Data  and  Analysis 


Graphs  for  individual  Services  are  presented  in  this  appendix.  The  data  and  re¬ 
sults  discussion  in  the  main  body  of  the  paper  are  repeated  verbatim  in  this  sec¬ 
tion,  following  each  applicable  graph. 

The  individual  Service  results  are  presented  for  each  of  the  following  metrics: 

♦  Output 

♦  Operating  Cost 

♦  Capital  Cost 

♦  Quality. 

For  each  metric,  the  results  are  listed  for  the  Army,  Air  Force,  NAVAIR, 
NAVSEA,  and  the  Marine  Corps,  respectively. 


Output 

The  following  graphs  present  the  output  metric  results,  in  terms  of  DLH. 

For  the  Army  (Figure  A-l),  a  substantial  (approximately  25  percent)  decrease  in 
workload  occurred  early  in  the  reported  period,  from  FY93-FY94.  This  is  not 
unexpected  due  to  the  significant  drawdowns  in  the  early  1990s.  The  transition 
from  DMOI  to  BES  data  is  not  smooth.  The  Army’s  subsequent  rise  in  DLH  is 
due  to  BES  data  that  includes  munitions  depot  maintenance,  which  is  not  included 
in  the  DMOI  data.  Since  the  BES  data  do  not  report  at  the  depot  level,  specific 
depots’  data  cannot  be  removed.  In  summary,  the  Army  DLH  trend  is  not  an  area 
of  concern. 
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Figure  A-l.  Direct  Labor  Hours— Army 


The  Air  Force’s  DLH  decreased  (Figure  A-2)  at  a  relatively  constant  rate  (ap¬ 
proximately  5  percent  per  year).  The  transition  point  from  the  DMOI  data  to  the 
BES  data  appears  to  be  smooth,  which  is  the  desired  behavior.  Unlike  the  Army 
data,  it  appears  that  the  Air  Force  DMOI  and  BES  data  are  directly  comparable. 
Additionally,  the  downward  trend  is  a  first-level  indicator  of  actions  being  taken 
to  reduce  DLH. 


Figure  A-2.  Direct  Labor  Hours— Air  Force 


NAVAIR  data  (Figure  A-3)  indicated  that  workload  increased  in  the  FY94-FY96 
time  frame  and  is  predicted  to  remain  constant  through  FY99.  The  transition  from 
DMOI  to  BES  data  is  smooth. 
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Detailed  Data  and  Analysis 


Figure  A-3.  Direct  Labor  Hours — NAVAIR 


The  NAVSEA  output  data  (Figure  A-4)  indicate  that  workload  is  decreasing  at  a 
consistent  rate  over  the  time  frame  plotted,  with  the  transition  between  DMOI  and 
BES  data  being  seamless. 

Figure  A-4.  Direct  Labor  Hours — NAVSEA 


The  USMC  (Figure  A-5)  data  indicate  that  workload  is  decreasing  at  a  slow,  yet 
constant  rate,  with  a  somewhat  larger  drop  planned  for  FY99.  Again,  the  transi¬ 
tion  from  DMOI  to  BES  also  is  smooth. 


A-3 


Figure  A-5.  Direct  Labor  Hours— Marine  Corps 


Operating  Cost 

The  two  operating  cost  metrics  were  Operating  expense/DLH  and  Overhead- 
cost/DLH.  Each  is  graphed  below  for  each  Service. 

For  the  Army  (Figures  A-6  and  A-7),  the  change  of  the  DLH  baseline  (the  BES 
data  include  munitions  depot  maintenance,  but  DMOI  data  do  not)  has  caused  the 
BES  data  to  be  generally  lower  than  the  DMOI  data.  The  Operating  ex¬ 
pense/DLH  and  Overhead  cost/DLH  indicate  that  the  Army  depots  do  well  in  an 
increasing-workload  environment  (FY97  to  FY98).  When  DLH  increases,  if  both 
overhead  and  DLC  increase  proportional  to  DLH,  the  Operating  expense/DLH 
curve  would  be  flat.  However,  since  the  curve  slopes  downward,  operating  ex¬ 
penses  are  increasing  at  a  slower  rate  than  DLH.  Similarly,  the  downward  sloping 
Overhead  cost/DLH  line  indicates  that  overhead  is  also  increasing  at  a  slower  rate 
than  DLH. 
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Detailed  Data  and  Analysis 


Figure  A-6.  Operating  Expense/DLH — Army 
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Trends  in  decreasing  workload  for  the  Army  depot  are  not  as  favorable.  Operat¬ 
ing  expense/DLH  does  not  increase  dramatically  in  FY98-FY99  when  DLH  in¬ 
creases.  This  indicates  that  operating  expenses  are  reduced  at  about  the  same  rate 
as  the  workload  decrease.  The  slight  growth  shown  by  the  Overhead  cost/DLH 
line,  in  combination  with  the  almost  flat  line  for  Operating  expense/DLH  indi¬ 
cates  that  overhead  is  decreasing  less  rapidly  than  DLH.  In  fact,  these  curves  in¬ 
dicate  that  DLC  is  potentially  decreasing  faster  than  DLH,  and  overhead  is 
becoming  the  more  substantial  component,  which  is  not  a  desired  outcome. 
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Overhead,  as  an  individual  data  element,  was  reported  only  for  a  subset  of  the  Air 
Force  depots  in  the  DMOI  (Figures  A-8  and  A-9).  Therefore,  the  FY96  Overhead 
cost/DLH  data  point  is  not  valid.  Focusing  on  the  FY97-FY99  time  frame,  DLH 
decreases  in  both  periods.  For  the  period  FY97-FY98,  Overhead  cost/DLH and 
Operating  expense/DLH  rise,  indicating  that  at  least  overhead  is  not  decreasing  at 
the  rate  of  DLH.  The  FY98-FY99  line  segments  indicate  that  overhead  begins  to 
decrease  more  rapidly  than  DLH,  but  DLC  does  not  decrease  as  fast  as  DLH.  The 
combination  of  these  two  effects  indicates  that  DLC  does  not  decrease  at  a  rate 
proportional  to  DLH. 

Figure  A-8.  Operating  Expense/DLH — Air  Force 


Detailed  Data  and  Analysis 


Figure  A-9.  Overhead  Cost/DLH — Air  Force 


For  NAVAIR  (Figures  A-10  and  A-l  1),  if  the  FY95  and  FY96  data  were  aver¬ 
aged,  the  trend  for  operating  expenses  and  Operating  expense/DLH  would  be  a 
decreasing  function — that  average  data  point  would  be  the  same  as  the  FY97  BES 
computed  point.  Because  Operating  expense/DLH  and  Overhead  cost/DLH  re¬ 
mains  constant  or  decreases  while  DLH  increases,  NAVAIR  is  doing  well  con¬ 
trolling  its  expenses. 

Figure  A-10.  Operating  Expense/DLH — NAVAIR 
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Figure  A-ll.  Overhead  Cost/DLH — NAVAIR 


The  NAVAIR  FY95  datapoint  departs  significantly  from  an  otherwise  consistent 
FY93  to  FY96  trend.  Further  investigation  found  that  the  operating  expense  re¬ 
ported  in  the  third  quarter  of  FY95  was  negative  for  all  three  NAVAIR  entities. 
The  data  will  not  be  changed;  however,  this  data  point  should  be  ignored. 

When  DLH  either  increases  or  decreases,  one  would  desire  both  Operating  ex- 
pense/DLH  and  Overhead  cost/DLH  to  decrease.  This  is  not  the  case  with  the 
NAVSEA  data  (Figures  A-12  and  A-13).  During  FY97-FY98,  Overhead 
cost/DLH  is  constant  but  Operating  expense/DLH  increases.  This  indicates  that 
DLCs  are  not  decreasing  at  the  same  rate  as  DLH,  which  means  that  there  may  be 
a  problem  with  DLG.  During  FY98-FY99,  Overhead  cost/DLH  increases,  indi¬ 
cating  that  overhead  is  not  decreasing  at  the  same  rate  as  DLH,  which  is  poten¬ 
tially  a  problem.  This  may  be  a  case  where  the  workload  may  be  constant  but 
labor  hours  are  just  being  reclassified  as  overhead  from  the  direct  category. 
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Figure  A-12.  Operating  Expense/DLH — NAVSEA 


Figure  A-13.  Overhead  Cost/DLH — NAVSEA 
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For  the  Marine  Corps  (Figures  A- 14  and  A- 15),  we  see  that  the  more  substantial 
drop  in  FY98-FY99  DLH  did  not  have  a  corresponding  decrease  in  overhead.  It 
appears  that  DLC  decreased  (at  least  proportionally  to  DLH),  but  overhead  did 
not  decrease  proportionately. 
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Figure  A- 14.  Operating  Expense/DLH — Marine  Corps 


Figure  A-15.  Overhead  Cost/DLH— Marine  Corps 
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Capital  Cost 

The  two  capital  cost  metrics  are  graphed  below,  for  each  Service. 

The  Army’s  (Figures  A-16  and  A- 17)  throughput  (revenue  less  direct  materials) 
increased  from  the  DMOI  data  to  the  BES  data  because  ordnance  depots  were  in¬ 
cluded  in  the  BES,  but  not  the  DMOI.  From  FY97-FY99,  Net  capital  investment 
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decreased.  With  this  in  mind,  the  data  indicate  that  while  not  replenishing  the 
capital  replacement  fund  (i.e.,  net  investment  decreased),  throughput  rose.  We 
cannot  isolate  the  contribution  to  throughput  that  is  new  mission  or  new  equip¬ 
ment  that  is  “free  issue”  to  the  depot;  however,  these  data  could  cause  one  to  in¬ 
vestigate  this  further. 

Figure  A- 16.  Capital  Effectiveness  Ratio — Army 


Fiscal  year 

Air  Force  throughput  increased  significantly  from  DMOI  to  BES  data  points 
(Figures  A-18  and  A-19).  We  verified  our  computations  but  could  not  uncover  a 
clear  reason  for  this  jump.  All  that  can  be  said  is  that  the  trend  is  an  increase  of 
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the  Capital  effectiveness  ratio,  even  in  years  of  negative  net  investment.  This  in¬ 
dicates  the  maintainability  or  improvement  of  throughput  with  a  negative  “capital 
flow,”  at  least  temporarily  in  the  time  frame  studied. 

Figure  A- 18.  Capital  Effective  Ratio — Air  Force 


Figure  A-19.  Net  Capital  Investment— Air  Force 


NAV AIR’s  capital  cost  performance  was  “middle  of  the  road”  (Figures  A-20  and 
A-21).  Throughput  tended  to  increase  with  low  net  investment.  However,  in  later 
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years,  throughput  did  not  increase  substantially  by  the  increased  positive  net  in¬ 
vestment. 


Figure  A-20.  Capital  Effectiveness  Ratio— NAV AIR 


Figure  A-21.  Net  Capital  Investment— NAVAIR 
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NAVSEA  did  not  show  promising  capital  cost  performance  (Figures  A-22  and 
A-23).  Generally,  throughput  decreased  while  capital  investment  was  consis¬ 
tently  negative.  The  resulting  indication  is  that  throughput  decreased  at  a  greater 
rate  than  capital  assets. 
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Figure  A-22.  Capital  Effectiveness  Ratio — NAVSEA 


Fiscal  year 


Figure  A-23.  Net  Capital  Investment — NAVSEA 


93  94  95  96  97  98 

Fiscal  year 

The  Marine  Corps’  performance  also  was  not  encouraging  (Figures  A-24  and 
A-25).  The  most  recent  data  indicate  a  decreasing  trend  in  throughput,  while 
capital  investment  hovered  around  zero.  The  observation  is  that  throughput  de 
creased  at  a  greater  rate  than  assets. 


Detailed  Data  and  Analysis 


Figure  A-24.  Capital  Effectiveness  Ratio — Marine  Corps 


Figure  A-25.  Net  Capital  Investment — Marine  Corps 


Quality 

The  quality  metric,  schedule  conformance,  results  are  shown  below  for  each 
Service. 
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The  Army’s  average  schedule  conformance  increased  from  FY95  to  FY96  (Figure 
A-26);  however,  with  only  two  data  points  available,  it  would  be  premature  to  call 
this  a  trend. 

Figure  A-26.  Schedule  Conformance— Army 


The  Air  Force  schedule  conformance  appears  to  have  increased  (Figure  A-27). 
However,  noting  the  scale  of  the  graph,  the  change  is  approximately  5  percent,  so 
quality  may  be  considered  constant  prior  to  and  following  the  jump  in  the  FY94- 
FY95  time  frame. 


A-16 


Detailed  Data  and  Analysis 


Figure  A-27.  Schedule  Conformance — Air  Force 


NAV AIR’s  schedule  conformance  decreased  from  FY93-FY96  by  approximately 
13  percent  (Figure  A-28).  This  metric  indicates  an  undesirable  trend  and  high¬ 
lights  an  area  for  improvement,  based  on  the  available  historical  data. 


Figure  A-28.  Schedule  Conformance — NAV  AIR 


As  discussed  above,  NAVSEA  and  the  Marine  Corps  do  not  report  schedule  data, 
since  their  end  items  and  logistics  systems  cause  such  data  to  be  misrepresentative 
of  true  depot  performance. 
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Appendix  B 

Abbreviations 


BES 

Budget  Estimate  Submission 

DLA 

Defense  Logistics  Agency 

DLC 

direct  labor  cost 

DLH 

direct  labor  hours 

DMOI 

Depot  Maintenance  Operations  Indicators 

DoD 

Department  of  Defense 

DWCF 

Defense  Working  Capital  Fund 

FMF 

Fleet  Marine  Forces 

G&A 

general  and  administrative 

LMI 

Logistics  Management  Institute 

MILCON 

military  construction 

NAVAR 

Naval  Air  Systems  Command 

NAVSEA 

Naval  Sea  Systems  Command 

OSD 

Office  of  the  Secretary  of  Defense 

POM 

Program  Objective  Memorandum 

RPM 

.  real  property  maintenance 

TE 

Table  of  Equipment 

USAF 

U.S.  Air  Force 

USMC 

U.S.  Marine  Corps 

WCF 

working  capital  fund 
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